Stochastic Majorization-Minimization Algorithms for Large-Scale Optimization

نویسنده

  • Julien Mairal
چکیده

Majorization-minimization algorithms consist of iteratively minimizing a majorizing surrogate of an objective function. Because of its simplicity and its wide applicability, this principle has been very popular in statistics and in signal processing. In this paper, we intend to make this principle scalable. We introduce a stochastic majorization-minimization scheme which is able to deal with largescale or possibly infinite data sets. When applied to convex optimization problems under suitable assumptions, we show that it achieves an expected convergence rate of O(1/ √ n) after n iterations, and of O(1/n) for strongly convex functions. Equally important, our scheme almost surely converges to stationary points for a large class of non-convex problems. We develop several efficient algorithms based on our framework. First, we propose a new stochastic proximal gradient method, which experimentally matches state-of-the-art solvers for large-scale l1logistic regression. Second, we develop an online DC programming algorithm for non-convex sparse estimation. Finally, we demonstrate the effectiveness of our approach for solving large-scale structured matrix factorization problems.

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Incremental Majorization-Minimization Optimization with Application to Large-Scale Machine Learning

Majorization-minimization algorithms consist of successively minimizing a sequence of upper bounds of the objective function. These upper bounds are tight at the current estimate, and each iteration monotonically drives the objective function downhill. Such a simple principle is widely applicable and has been very popular in various scientific fields, especially in signal processing and statist...

متن کامل

On the Global Convergence of Majorization Minimization Algorithms for Nonconvex Optimization Problems

In this paper, we study the global convergence of majorization minimization (MM) algorithms for solving nonconvex regularized optimization problems. MM algorithms have received great attention in machine learning. However, when applied to nonconvex optimization problems, the convergence of MM algorithms is a challenging issue. We introduce theory of the KurdykaLojasiewicz inequality to address ...

متن کامل

An Ant Colony Optimization Algorithm for Network Vulnerability Analysis

Intruders often combine exploits against multiple vulnerabilities in order to break into the system. Each attack scenario is a sequence of exploits launched by an intruder that leads to an undesirable state such as access to a database, service disruption, etc. The collection of possible attack scenarios in a computer network can be represented by a directed graph, called network attack gra...

متن کامل

Low-Rank Doubly Stochastic Matrix Decomposition for Cluster Analysis

Cluster analysis by nonnegative low-rank approximations has experienced a remarkable progress in the past decade. However, the majority of such approximation approaches are still restricted to nonnegative matrix factorization (NMF) and suffer from the following two drawbacks: 1) they are unable to produce balanced partitions for large-scale manifold data which are common in real-world clusterin...

متن کامل

Solving single facility goal Weber location problem using stochastic optimization methods

Location theory is one of the most important topics in optimization and operations research. In location problems, the goal is to find the location of one or more facilities in a way such that some criteria such as transportation costs, customer traveling distance, total service time, and cost of servicing are optimized. In this paper, we investigate the goal Weber location problem in which the...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

عنوان ژورنال:

دوره   شماره 

صفحات  -

تاریخ انتشار 2013